Categories

Versions

Tukey Test (Operator Toolbox)

Synopsis

This operator finds outliers according to the Tukey Test.

Description

The Tukey Test is defined for numerical type attributes. For each selected attribute a confidence of the Tukey Test is calculated. This confidence is defined as the distance between the current value to the median, divided by the distance of the lower/upper 'Tukey Test boundary' to the median. The lower/upper 'Tukey Test boundary' is calculated as [Q_1 - 1.5*IQR,Q_3 + 1.5*IQR]. If the confidence is greater than or equal to one, the example is considered to be a 'Bottom'/'Top' Outlier. A new nominal attribute with ('No Outlier','Bottom Outlier','Top Outlier') values is created. Also the calculated confidences is written to a new attribute. In addition, an attribute 'TukeyTest_NumberOfOutlierAttributes' is created, which contains for each example how many attributes are considered to have outlier values.

The operator provides the Tukey Test Model at the preprocessing model outputport. It can be used to apply the Tukey Test on unseen data with the Apply Model operator.

Input

  • example set (Data Table)

    The input ExampleSet.

Output

  • example set output (Data Table)

    The resulting output ExampleSet with the additional Tukey Test attributes.

  • original (Data Table)

    The original ExampleSet.

  • preprocessing model

    The Tukey Test Model as a preprocessing model. It can be used to be applied on new data with the Apply Model operator.

Parameters

  • attribute_filter_type

    Only attribute with type numerical can be selected for the Tukey Test. This parameter allows you to select the attribute selection filter; the method you want to use for selecting attributes. It has the following options:

    • all: This option selects all the attributes of the ExampleSet, no attributes are removed. This is the default option.
    • single: This option allows the selection of a single attribute. The required attribute is selected by the attribute parameter.
    • subset: This option allows the selection of multiple attributes through a list (see parameter attributes). If the meta data of the ExampleSet is known all attributes are present in the list and the required ones can easily be selected.
    • regular_expression: This option allows you to specify a regular expression for the attribute selection. The regular expression filter is configured by the parameters regular expression, use except expression and except expression.
    • value_type: This option allows selection of all the attributes of a particular type. It should be noted that types are hierarchical. For example real and integer types both belong to the numeric type. The value type filter is configured by the parameters value type, use value type exception, except value type.
    • block_type: This option allows the selection of all the attributes of a particular block type. It should be noted that block types may be hierarchical. For example value_series_start and value_series_end block types both belong to the value_series block type. The block type filter is configured by the parameters block type, use block type exception, except block type.
    • no_missing_values: This option selects all attributes of the ExampleSet which do not contain a missing value in any example. Attributes that have even a single missing value are removed.
    • numeric_value_filter: All numeric attributes whose examples all match a given numeric condition are selected. The condition is specified by the numeric condition parameter.
    Range:
  • attribute

    The required attribute can be selected from this option. The attribute name can be selected from the drop down box of the parameter if the meta data is known.

    Range:
  • attributes

    The required attributes can be selected from this option. This opens a new window with two lists. All attributes are present in the left list. They can be shifted to the right list, which is the list of selected attributes that will make it to the output port.

    Range:
  • regular_expression

    Attributes whose names match this expression will be selected. The expression can be specified through the edit and preview regular expression menu. This menu gives a good idea of regular expressions and it also allows you to try different expressions and preview the results simultaneously.

    Range:
  • use_except_expression

    If enabled, an exception to the first regular expression can be specified. This exception is specified by the except regular expression parameter.

    Range:
  • except_regular_expression

    This option allows you to specify a regular expression. Attributes matching this expression will be filtered out even if they match the first expression (expression that was specified in regular expression parameter).

    Range:
  • value_type

    This option allows to select a type of attribute. One of the following types can be chosen: nominal, numeric, integer, real, text, binominal, polynominal, file_path, date_time, date, time.

    Range:
  • use_value_type_exception

    If enabled, an exception to the selected type can be specified. This exception is specified by the except value type parameter.

    Range:
  • except_value_type

    The attributes matching this type will be removed from the final output even if they matched the before selected type, specified by the value type parameter. One of the following types can be selected here: nominal, numeric, integer, real, text, binominal, polynominal, file_path, date_time, date, time.

    Range:
  • block_type

    This option allows to select a block type of attribute. One of the following types can be chosen: single_value, value_series, value_series_start, value_series_end, value_matrix, value_matrix_start, value_matrix_end, value_matrix_row_start.

    Range:
  • use_block_type_exception

    If enabled, an exception to the selected block type can be specified. This exception is specified by the except block type parameter.

    Range:
  • except_block_type

    The attributes matching this block type will be removed from the final output even if they matched the before selected type by the block type parameter. One of the following block types can be selected here: single_value, value_series, value_series_start, value_series_end, value_matrix, value_matrix_start, value_matrix_end, value_matrix_row_start.

    Range:
  • numeric_condition

    The numeric condition used by the numeric condition filter type. A numeric attribute is kept if all examples match the specified condition for this attribute. For example the numeric condition '> 6' will keep all numeric attributes having a value of greater than 6 in every example. A combination of conditions is possible: '> 6 && < 11' or '<= 5 || < 0'. But && and || cannot be used together in one numeric condition. Conditions like '(> 0 && < 2) || (>10 && < 12)' are not allowed because they use both && and ||. Nominal attributes are always kept, regardless of the specified numeric condition.

    Range:
  • include_special_attributes

    Special attributes are attributes with special roles. These are: id, label, prediction, cluster, weight and batch. Also custom roles can be assigned to attributes. By default all special attributes are delivered to the output port irrespective of the conditions in the Select Attribute operator. If this parameter is set to true, special attributes are also tested against conditions specified in the Select Attribute operator and only those attributes are selected that match the conditions.

    Range:
  • invert_selection

    If this parameter is set to true the selection is reversed. In that case all attributes matching the specified condition are removed and the other attributes remain in the output ExampleSet. Special attributes are kept independent of the invert selection parameter as along as the include special attributes parameter is not set to true. If so the condition is also applied to the special attributes and the selection is reversed if this parameter is checked.

    Range:
  • ignore_missing_values

    If this parameter is set to true missing values are ignored in the computations of the median and quartiles which are used to define the Tukey Test confidence.

    Range:

Tutorial Processes

Tukey Criterion on Sonar Data

This Process retrieves the Sonar data and performs the Tukey Test for the attributes: attribute_1, attribute_2 and attribute_3. The TukeyTest_NumberOfOutlierAttributes is set to the special role 'n', so that is ordered in the result view at the beginning. The data set is sorted decreasingly according to the TukeyTest_NumberOfOutlierAttributes, so that the examples with outliers are on top.